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Introduction 


In January 2012, the West Virginia Governor’s Office released an influential report, Educa- 
tion Efficiency Audit of West Virginia’s Primary and Secondary Education System, written by 
Public Works (2012), a management consulting company headquartered in West Chester, 
PA. Based largely on this report, the West Virginia Legislature passed education reform leg- 
islation, which was signed into law by the Governor in spring 2013. 

The Catalyst School Research Study examined the initial stage of an initiative that grew out 
of that reform. The Transforming Professional Learning initiative focused on changing the 
way the West Virginia Board of Education (WVBE) defined professional learning and the 
ways major providers involved themselves in the professional learning of educators in the 
state, especially the West Virginia Department of Education (WVDE) and regional education 
service agencies (RESAs). 

The results of the study were reported in five brief reports, all available at 

http://wvde.state.wv.us/research/reports 2016 .html : 

• Catalyst Schools Research Study Report: Supporting Statewide Implementation of 
the Learning School Initiative, May 201 6 

• Catalyst Schools Research Study Report: Spotlight on RESA 2 - Regional 
Implementation of the Learning School Initiative, May 2016 

• Catalyst Schools Research Study Report: Spotlight on RESA 6 - Regional 
Implementation of the Learning School Initiative, May 2016 

• Catalyst Schools Research Study Report: Catalyst Schools’ Implementation of the 
Learning School Approach, July 201 6 

• Catalyst School Research Study: The Learning School Approach and Student 
Proficiency in ELA and Math: Preliminary Findings, September 201 6 

This report provides technical details about the research methods used in the first four re- 
ports in the study. The fifth report includes methods in the report itself. 

Background 

The WVBE High Quality Educator (HQE) Committee obtained funding from the Claude 
Worthington Benedum Foundation in mid-2013 to support the Transforming Professional 
Development project. The Board contracted with the National Commission for Teaching and 
America’s Future (NCTAF) to lead it, with other consulting help from Learning Forward, the 
National Governors Association, and the U.S. Education Delivery Institute. The overarching 
principle of the project was to redefine educators’ conception of schools as communities of 
learners, including students, staff, and parents. Further, the system of professional learning 
would shift from a provider-driven perspective — where state agencies, RESAs, and others 
developed a slate of offerings from which individual educators could make selections to fulfill 
their professional development requirements — to a system driven by a cycle of continuous 
improvement in which educators collaboratively determined their learning needs based on 
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the learning needs of students in their schools. RESAs’ role in the system would be to pro- 
vide technical assistance and training to support this shift to school- and district-based pro- 
fessional learning communities (PLCs). 

To launch this new approach and to model various ways professional learning can be driven 
at the local level, leadership of the initiative identified two RESAs (2 and 3) to work in part- 
nership with two or three schools in their regions as catalyst schools — that is, as prototypes 
with the added credibility of emerging in West Virginia’s own schools. Catalyst schools were 
chosen based on their readiness to engage in the project, using the following criteria (Killion, 
2014b, pp. 3-4): 

• Strong school-level leadership support; 

• Support from RESA and county leadership; 

• Established and well-functioning PLCs; 

• Understanding of elements of effective professional learning as documented in the 
research; 

• Interest in exploring how to reallocate time and resources; 

• Strong relationship with RESA leadership; 

• Willingness to be transparent about the process, funding, challenges 

Cohort 1 included five schools in three counties. A second cohort of schools included 23 
schools, with one or more schools in each of the eight RESA regions. Cohort 2 began its 
participation during the summer of 2015 and continued, along with Cohort 1 , through the 
2015-2016 school year. 

Rationale for the Study 

The rationale for this study is grounded in its potential significance for informing RESAs, 
WVDE, and others about how a new, more cohesive and user-driven system of professional 
learning can be established across the state. 

Relevant Scholarship 

Professional development, as part of a reform strategy is only one element in creating an in- 
structional system that improves both teacher and student learning. Based on their studies 
of improving mathematics instruction, Cobb and Jackson (2011, p. 12) propose the ele- 
ments outlined below be included in a coherent instructional system. Many of these ele- 
ments are being developed in West Virginia, especially as part of the statewide 
implementation of the West Virginia College and Career Ready Standards. Following each 
of the elements (shown in italics) recent West Virginia initiatives and reforms are listed. 

• Explicit goals for students’ learning — Adoption of the West Virginia College and 
Career Readiness Standards 

• A detailed vision of high-quality instruction that specifies particular instructional 
practices that will lead to students’ attainment of the learning goals — Instructional 
shifts and Standards for Mathematical Practice 
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• Instructional materials and associated tools designed to support teachers’ 
development of these practices — The West Virginia TREE website 

( https://wvde.state.wv.us/apps/tree/) 

• District teacher professional development that focuses on the specific practices, 
is organized around the above materials, and is sustained over time — Train-the- 
trainer events such as the Educator Enhancement Academies and 2016 
institutes, which trained teachers and district central office staff in the West 
Virginia College and Career Ready Standards and instructional shifts 

• School-based PLCs that provide ongoing opportunities for teachers to discuss, 
rehearse, and adapt the practices that have been introduced in district 
professional development — The Learning School initiative 

• Classroom assessments aligned with the goals for students’ learning that can 
inform the ongoing improvement of instruction and the identification of students 
who are currently struggling — New interim assessments aligned with the West 
Virginia College and Career Readiness Standards 

• Additional supports for struggling students to enable them to succeed in 
mainstream classes — Supports for Personalized Learning and Supports for 
Personalized Instruction 

The Learning School initiative — including its pilot-project catalyst schools which are the ob- 
ject of this study — focus largely on the fifth bullet in this list. See the discussion section of 
the fourth report in this series, Catalyst Schools’ Implementation of the Learning School Ap- 
proach, for additional discussion of the research literature on PLCs, and the Standards for 
Professional Learning. 

Evaluation Questions 

The purpose of the catalyst schools was to demonstrate and document a new, school- 
based, student-needs-driven approach to professional development that can then be scaled 
up and implemented broadly across the state. This evaluation created a framework for inter- 
preting the experiences that unfolded at the catalyst schools, to provide insights and data for 
consideration in the broader implementation of this new system for professional learning. 

The evaluation focused on the following broad questions: 

EQ1 . What are feasible approaches for statewide implementation of the 
Learning School approach to professional learning? 

EQ2. What services/products were provided to support implementation in 
the schools? 

EQ3. How did implementation take shape in the schools, especially with 
regard to alignment with the Learning Forward Standards for 
Professional Learning? 

EQ4. To what extent did the project support professional learning that 
improves educator effectiveness? 
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Methods 

This evaluation study employed both quantitative and qualitative methods to provide usea- 
ble information for leaders in the catalyst schools, as well as for planners and leaders in- 
volved in scaling up the program for statewide implementation. Stakeholders were involved 
in data collection and interpretation in order to maximize the usefulness and use of the study 
findings. All focus group and individual interviews were recorded and transcribed and read 
multiple times to identify themes related to the evaluation questions. Additional details are 
included in the descriptions of the individual reports below. For a brief timeline for the study, 
see Appendix A. The remainder of this technical report is organized by the four reports de- 
scribed earlier, which collectively address the evaluation questions listed above. 

This study was approved by the WVDE Institutional Review Board (IRB-WVDE-027). 

Supporting Statewide Implementation of the Learning School Initiative 

This report examined three topics: funding decisions, scheduling time, and capacity building. 
Data sources and methods of analysis are described for each topic. 

Funding decisions 

• Information about funding, including restrictions on its use, was obtained in an 
interview with staff from the WVDE Office of Federal Programs on April 15, 2016 
(see Appendix B). 

• Information about variations among districts in their responsiveness to catalyst 
school funding requests was obtained during focus group interviews with Cohorts 1 
and 2 catalyst school principals at all eight RESAs from November 30, 2015 through 
January 8, 2016 (see Items 5-7, Appendix C). Focus groups were recorded, 
transcribed, and coded using a qualitative analysis application, RQDA. Additional 
information was obtained in conversations with staff from RESAs 2 and 6 during the 
winter of 2016. 

Scheduling time 

• Information in this report about scheduling time was obtained in focus group 
interviews with Cohort 1 catalyst school teachers in the spring of 2015 (see Appendix 
D); and with Cohorts 1 and 2 principals at all eight RESAs from November 30, 2015 
through January 8, 2016 (see Appendix C). 

Capacity building 

• As a member of the Transforming Professional Development Advisory Group 
beginning in 2013, I was able to provide this account based on my own notes about 
the progression of events from the beginning of the project. For a more detailed 
timeline, see Appendix E. 
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Spotlight on RESA 2: Regional Implementation of the Learning School Initiative 

This case study examined the approach used by one of the RESAs, selected because of the 
observed, strong performance of its catalyst schools overall. 

• RESA 2 supported three elementary schools in the first cohort of catalyst schools 
and added five more schools for the second cohort. Due to this early involvement, 
former Executive Director Dee Cockrille and Coordinator of Curriculum and 
Instruction Tammy Stowers made themselves available for multiple interviews and to 
answer questions beginning in May 2015 (see Appendix F) through to the end of 
data collection in the spring of 2016. 

• Quotations from principals were drawn from two sources, a focus group interview 
conducted on December 1 8, 201 5 at the RESA 2 facility (see Appendix C for the 
interview questions) and comments in response to an open-ended question on a 
PLC scheduling survey begun on March 2, 2016 (see Appendix G for the survey 
invitation and form). 

• Dr. Cockrille and Ms. Stowers responded to a detailed questionnaire in early March 
2016 (see Appendix H). 

Spotlight on RESA 6: Regional Implementation of the Learning School Initiative 

This case study examined the approach used by one of the RESAs, selected because of the 
observed, strong performance of its catalyst schools overall. 

• RESA 6 joined the catalyst school pilot project in May 201 5 with four catalyst 
schools. Data collection for this RESA began with the principals’ focus group, 
conducted at RESA 6 on January 6, 2016. Similarly to RESA 2, quotations from 
principals were pulled from this interview (Appendix C) and from questions on a PLC 
scheduling survey mentioned above (Appendix G). 

• Additionally, I participated in an extended discussion on February 26, 2016, during 
which RESA 6 Executive Director Nick Zervos, and staffers Marian Kajfez, Mary Kay 
Reisinger, Michelle Plogan, and Jon Pollack explained the RESA 6 school 
improvement process and provided me with catalyst school scheduling information 
and many other materials related to their approach to regional support and their work 
with school districts. 

• I attended the March 2016 RESA 6 Regional Principals Institute, which featured 
presentations by catalyst school principals and teacher leaders explaining their 
implementation of the Learning School approach in their schools to a regional 
audience of principals and district administrators. 

Catalyst Schools’ Implementation of the Learning School Approach 

To develop a clearer picture of the variation among catalyst schools in their implementation 
of the Learning School approach, I constructed a 5-point index using four measurements for 
each individual school. Using the four measurements — or in some cases three when all four 
were not available — I then calculated a “level of implementation rating” for each catalyst 
school. The measurements included 
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1 . Time allotted in schedules for PLCs 

2. Types of activities included in PLC agendas 

3. Standard Assessment Inventory 2 (SAI2) scores 

4. RESA and the WVDE assessments of each school’s stage of implementation 

Each of these measurements also used a 5-point scale, so the composite index score was 
calculated by taking the mean of the scores. Schools for which there were fewer than three 
of the measurements available, were not included in the analysis. 

Focus group interviews with principals were also used to augment the SAI2 scores, for rea- 
sons explained below. 

Time allotted in schedules for PLCs 

In all of my interviews with principals and teachers, I asked about how much time they had 
for their PLCs and how they managed to schedule it. However, interview transcripts were 
difficult to use for determining, specifically, how much time each of the schools had sched- 
uled for their individual PLCs to meet. Consequently, on March 2, 201 6, I asked principals to 
send me their list of PLCs, and indicate how often each one typically met, for how many 
minutes, and a few details about how they scheduled the time (see Appendix G for the sur- 
vey invitation and form). Twenty-five of the 28 catalyst schools responded to this request. 

Standardized responses were set up to al- 
low for the collection of data about fre- 
quency and duration of meetings, which 
could be compared across schools (see 
box for response options). Based on the 
frequency and duration questions and a 4- 
week month, I was able to calculate for 
each school, the average number of 
minutes per month each PLC typically met. 

Then I took the average for each school. 

As described in the report of findings, I 
was able to compare this average to the 
number of minutes called for in the litera- 
ture — about 60 minutes in an 8-hour day. 

The top of this scale (5 points) approaches 
this amount of time. I cut the number of 
minutes in half for each descending level 
on the scale, as follows: 

5 = 720+ (minimum = 4 days/week @ 45 
minutes/day or more) 

4 = 360 - 719 (minimum = 2 days/week @ 

45 minutes/day) 

3 = 180 - 359 (minimum = 1 day/week @ 

45 minutes/day) 


Response Options for PLC Frequency 
and Duration Questions 

Usual frequency of meetings (select one) 

o 1 time/every 2 months 
o 1 time/month 
o 2 times/month 
o 3 times/month 
o 1 time/week 
o 2 times/week 
o 3 times/week 
o 4 times/week 
o 5 times/week 

When are meetings held? (select one) 

o During work day before students arrive 
o During work day after students leave 
o During work day with coverage by 
substitute(s) 

o During work day with coverage by other 
staff 

o During work day when students are off 
o Outside of regular work hours 
o Other 
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2 = 90 - 179 (minimum = 2 day/month @ 45 minutes/day) 

1 =0-89 (less than 2 day/month @ 45 minutes/day) 

After each school was assigned a scale score for the amount of time scheduled for PLCs, 
the scores were aggregated and averaged by RESA and across the 25 catalyst schools that 
provided information. 

Types of activities included in PLC agendas 

Eighteen schools responded to a request for copies of their PLC agendas for the month of 
February 2016. Some catalyst schools reported weather-related cancellations of February 
PLCs, so they sent late January and early March agendas. Agendas from outside of this 
narrow window were not included in the analysis. Additionally, one school sent along two 
faculty meeting agendas, which also were not included in the analysis. 

The agendas were anonymized and schools were randomly assigned pseudonyms. Then 
agenda items were numbered continuously by individual school. For example, School M 
sent agendas for its nine PLCs. The agenda items were numbered continuously, starting 
with the first item on the first agenda and continuing through the remaining agendas, reach- 
ing a total of 91 numbered items. The range of numbered items across the schools was from 
five to 91 items. 

I developed in collaboration with the WVDE Office of Student and School Support, a coding 
scheme with definitions for categorizing the items on the agendas (for the complete list of 
codes and their definitions, see Appendix I). The three major coding categories were (a) ad- 
ministrative information sharing, (b) planning for learning, and (c) engaging in professional 
learning. The major categories had three or four subcategories, each with its own code. 

A RESA allowed one of its school improvement coordinators to assist me with conducting 
the coding. After reviewing the codes and their definitions, the coordinator and I individually 
categorized the agenda topics for each school separately, entering the information into a 
spreadsheet with one worksheet for each school (see Appendix J). The columns on each 
worksheet were for the codes and the rows were for the numbered agenda items. Each 
agenda item was coded by entering a 1 in the column representing the appropriate code; 
agenda items were assigned only one code each. After independently coding each school, 
we compared our codes, item by item. Where our coding varied, we each described our rea- 
sons for assigning the codes we did, and then reached consensus on how to ultimately code 
the item. We proceeded this way through all 1 8 sets of school PLC agendas. 

For each school, I tabulated raw counts for each of the three major categories, and calcu- 
lated the major categories’ percentages of the total agenda items. At this point, I was ready 
to convert the percentages into a quality scale score for each school’s collection of agendas. 
I began by devising a weighting scheme that rewarded agenda items categorized as “en- 
gaged in professional learning” because they are the most desirable activities to be taking 
place during PLCs (see the discussion section of the report in this series titled Catalyst 
Schools’ Implementation of the Learning School Approach for an explanation of the re- 
search supporting this and other research-based assertions). 
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Agenda items focused on professional learning were weighted 1 .5 points for each percent- 
age point of agenda items. Based on the assumption that there is some degree of learning 
and sharing that takes place when collaborative teams get together to review student data 
and make plans for student learning, 1 point was assigned for each percentage point of 
planning items. A negative one quarter point (-.25) was assigned for administrative tasks 


more appropriately 
handled in staff meet- 
ings. Under this 
scheme, the lowest 
possible score was -25 
points, which would 

Table 1 . 

PLC Agenda Quality Measures 




Percentage of agenda items 

Administra- Engaged in 

tive informa- Planning for professional 
tion sharing learning learning 

Weighted 

value 

Scale 

score 

represent 1 00% of 

School A 

0.0 

0.0 

100.0 

150.0 

5 

agenda items falling in 

School C 

45.9 

37.8 

16.2 

73.6 

2 

the Administrative cate- 

School D-E 

61.9 

9.5 

28.6 

67.9 

2 

gory; the highest possi- 

School F 

26.3 

57.9 

15.8 

88.2 

3 

ble score would be 150 

School G 

28.6 

57.1 

14.3 

85.7 

3 

points, where 1 00% of 

School H 

3.3 

93.3 

3.3 

99.2 

3 

the PLC agenda items 

School 1 

14.3 

71.4 

14.3 

96.4 

3 

fell into the “Engaged in 

School J 

0.0 

62.5 

37.5 

118.8 

4 

professional learning” 

School K 

3.4 

86.2 

10.3 

102.6 

4 

category. Actual results 

School L 

0.0 

100.0 

0.0 

100.0 

3 

are shown in Table 1 . 

School M 

20.9 

68.1 

11.0 

89.8 

3 


School N 

13.2 

86.8 

0.0 

90.1 

3 

School O 

63.6 

27.3 

9.1 

56.8 

2 

School P 

14.3 

71.4 

14.3 

96.4 

3 

School Q 

39.0 

57.6 

3.4 

72.5 

2 

School S 

0.0 

0.0 

100.0 

150.0 

5 

School T 

46.2 

50.0 

3.8 

67.3 

2 

School U 

9.1 

90.9 

0.0 

93.2 

3 


The final stage involved converting these weighted scores to a 5-point scale. The scale is 
based on ranges of the weighted values, as shown in Table 2. Scale scores for schools 
were aggregated and averaged by RESA and overall across the 18 schools. 


8 


Methods 


Table 2. Relationship of Scale Scores to Weighted Values 
Weighted 
Scale value 

score range Description 


5 

4 

3 

2 

1 


126-150 At least 90% of items focused on professional learning and planning for learning combined, with 
vast majority of items focused on professional learning 

101 - 125 At least 90% of items focused on professional learning and planning for learning combined, with 
emphasis on planning 

76-100 At least 70% of items focused on professional learning and planning combined, with emphasis on 
planning, and a growing number of items focused on administrative tasks 

51 - 75 60% or less of items focused on professional learning and planning combined, with emphasis on 
planning and almost 40% of items focused on administrative tasks 


-(25) - 50 4% or more of items focused on administrative tasks, with few or no items focused on professional 
learning 


SAI2 scores 

The SAI2 is a valid and reliable instrument developed by Learning Forward to measure the 
extent to which individual schools align their professional learning practices with the Stand- 
ards for Professional Learning (Denmark & Weaver, 2012). Catalyst schools were invited to 
participate in the SAI2 for a pretest in the spring of 2015 and again for a posttest in the 
spring of 2016 (see Appendix G). Twenty-five of the 28 catalyst schools participated in the 
pretest, but only ten schools participated in the posttest survey by the end of June 2016. Be- 
cause of the low level of participation and the lack of representation across all RESAs, I was 
not able to provide a growth analysis that would have been representative of the catalyst 
schools’ experience overall. I did use the 2016 scores available for the 10 schools as one of 
four measures to construct the levels of implementation scale (see below). 

Focus group interviews with principals 

The focus group interviews with principals were conducted at all eight RESAs from Novem- 
ber 30, 201 5-January 8, 201 6 (see Appendix C). I had previously drawn upon transcripts 
from the interviews to learn about how the catalyst schools were supported by districts and 
RESAs. These interviews proved useful, once again, in addressing the evaluation question, 
“How did implementation take shape in the schools, especially with regard to alignment with 
the Learning Forward Standards for Professional Learning?” Here I wanted to take a closer 
look at what the principals had to say about specific actions they have taken to address the 
seven Standards for Professional Learning (see box below). 

To quantify their responses to my direct questions during the interviews about what actions 
they had taken to implement any of the seven standards, I used the rapid identification of 
themes from audio recordings (RITA), developed by Neal, Neal, VanDyke, and Kornbluh 
(2015). This method calls for coding the recorded interviews while listening to them, to iden- 
tify themes. To do this I set up a spreadsheet (see Appendix K) with 3-minute increments as 
the columns, and the seven standards and subdomains as identified by Learning Forward 
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(2011) as the rows (see the accompanying box). To provide a more granular system of cod- 
ing, I used the subdomains — three for each standard — instead of the standards themselves 
to do the coding, then summed the subdomains to obtain a total for each of the seven stand- 
ards. As explained by Neal, Neal, VanDyke, and Kornbluh (2015), because the time seg- 
ment is the recording unit, I only determined if the theme (subdomain) was mentioned within 
each particular time segment, and not how often it occurred. 


Standards for Professional Learning With Subdomains 


DATA — Uses to plan, assess, evaluate profes- 
sional learning 

• Student data (formal/informal 
assessments) 

• Educator data (educator evaluation, 
SAI2, etc.) 

• System data (OEPA, fiscal, HR, time, 
etc.) 

IMPLEMENTATION 

• Applies research on change 

• Sustains support for long-term change 

• Provide constructive feedback 

LEADERSHIP-for professional learning, includ- 
ing 

• Develops capacity 

• Advocates 

• Creates systems/structures (calendars, 
schedules, etc.) 

LEARNING COMMUNITIES-committed to 

• Continuous improvement 


• Collective responsibility 

• Goal alignment 

LEARNING DESIGNS 

• Aligns with teacher experience/needs 

• Uses variety of forms 

• Uses peer observation 

OUTCOMES 

• Explicitly links educator and student 
learning 

• Focuses on curriculum and how 
students learn 

• Builds on previous teacher learning 

RESOURCES 

• Allocates time for professional learning 
during school day 

• Expenditures for professional learning 
openly discussed 

• Provides technology and other 
professional learning resources 

Adapted from Learning Forward (2011) 


It should also be noted that because the interviews varied somewhat in length — from about 
40 minutes to over an hour — I used mentions of each of the standards as percentages of all 
mentions, instead of as totals. I made this choice because principals at one RESA may have 
had more time to mention a theme (standard) than at another RESA. 

This was important because I did want to make some comparisons across the interviews. 
The focus groups occurred at each of the eight RESAs and involved only the catalyst school 
principal(s) from the RESA where the interview occurred. Consequently, I could note differ- 
ences in the themes across the RESAs. 

I did not attempt to convert this analysis into the 5-point scale I used for the others because 
it was essentially measuring the same thing as the SAI2 — alignment of professional learning 
practices with the Standards for Professional Learning — and the SAI2 is a more precise 
measure. However, due to the lack of participation in the SAI2 in spring of 2016 (only 10 of 
the 28 schools), this seemed a useful supplement to those data. The results were presented 
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as prevalence of themes by RESA and overall, which gave some indication of standards 
that are well established and those that may need more attention. 

RESA and WVDE assessments of each school’s stage of implementation 

The fifth analysis drew upon implementation research, especially the seminal research syn- 
thesis of this body of research conducted by Fixsen, Naoom, Blase, Friedman, and Wallace 
(2005). These scholars identified five stages of implementation that interventions tend to 
pass through (see box below). 

I drew upon the familiarity of the RESA staff associated with each of the catalyst schools 
during the 2015-201 6 implementation period, and the WVDE staff member who, beginning 
in January 2016, had made at least one daylong site visit with each of the schools to ob- 
serve classrooms, attend PLCs, and meet with principals. 

I provided the stages-of-implementation definitions to RESA staff and asked them to assign 
the appropriate stage of implementation to each of their catalyst schools (see Appendix L). I 
made a similar request of the WVDE staff member on June 7, 2016. 

The WVDE ratings tended to run about 40% higher than the RESA scores overall. Also the 
WVDE ratings were frequently out of 
alignment with the other data points 
(SAI2, PLC agenda quality, and PLC 
duration). Further, in most cases, RE- 
SAs had more contact with the cata- 
lyst schools and training in the 
Learning School approach and in the 
Standards for Professional Learning. 

Consequently, I adjusted for this lack 
of inter-rater reliability by giving more 
weight to the RESA ratings. I multi- 
plied WVDE scores by .75 and the 
RESA scores by 1 .25. I then took the 
average of the two scores. 

The five stages of implementation 
converted easily to the 5-point scale I 
had employed in three of the other 
analyses. 

Composite rating 

The final analysis in this study combined 
the four scale scores— that is (a) time al- 
lotted in schedules for PLCs, (b) types of 
activities included in PLC agendas, (c) 

SAI2 scores, and (d) RESA and WVDE as- 
sessments of the stage of implementation in each of the catalyst schools— into an index score by 
taking average scores at the RESA level and across all catalyst schools. I had at least three scores 


Stage of Implementation 

Stage 1. Exploration— identifying the need, acquir- 

ing information, assessing the fit between the need and 
intervention program, and preparing the organization by 
mobilizing information and support 

Stage 2. Installation — making preparations, such as 

human resource strategies (e.g., scheduling time), pol- 
icy development, reporting frameworks, outcome ex- 
pectations, staff training, and needed 
technology/resources 

Stage 3. Initial implementation— beginning to use 
and monitor new practices, developing and applying 
new skills, creating a supportive organizational culture, 
and overcoming inertia and resistance 

Stage 4. Full implementation — proceeding with in- 
novation as accepted practice and routine, and seeing 
evidence of expected impacts/improvements 

Stage 5. Sustainability — maintaining the effective- 

ness of the innovation as staff come and go, and other 
changes that take place in the organization and its envi- 
ronment 

Adapted from Fixsen, Naoom, Blase, Friedman, and 
Wallace (2005). 
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for 18 of the catalyst schools, which happened to be distributed across all eight RESAs, al- 
lowing me to provide an average implementation score for every RESA and for the catalyst 
schools statewide. 

This composite, or index score removed some of the variation found in the individual anal- 
yses. As noted in the report, the two top scorers on the index were the two RESAs that were 
highlighted in the earlier case study reports, Spotlight on RESA 2 and Spotlight on RESA 6. 
The catalyst schools in these two regions were among the highest scorers on multiple quali- 
tative measures. 

Limitations of the Study 

The SAI2, an instrument that was validated by a third party research organization (Denmark 
& Weaver, 2012) was used in this study because it has been shown to be predictive of stu- 
dent performance at the school level. The other measures used relied on qualitative meth- 
ods that work best in combination with each other when making an assessment of the level 
of implementation at the school or RESA level. 

The results of this research study as reported in the four reports (Hammer, 2016a; 2016b; 
2016c; and 201 6d) should not be used summatively, however. The purpose of the study 
was to provide feedback to the WVBE, WVDE, and RESAs about the successes and chal- 
lenges in the implementation of the Standards for Professional Development as they have 
played out during the first two years of the implementation of the Learning School initiative. 
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Appendix A. Timeline for Study 

The evaluation of the implementation and outcomes of the catalyst school project took place 
according to the following brief timeline: 


2015 Activities 


February-March 

Data collection at catalyst schools using interview and focus group 
protocols for teachers and principals, SAI2 survey of teachers 

April-May 

Data analysis, writing 

July 

Administration of SAI2 pretest for teachers in Cohort 2 catalyst 
schools 

Participation in three Cohort 2 orientations for district staff and catalyst 
school leadership teams (northern and southern orientations) 

Individual school reports and aggregate reports, prepared as Power- 
Point presentations for Cohort 1 

July-November 

Reports of preliminary findings and brief handouts for the WVBE High 
Quality Educator Committee and Professional Learning Advisory 


Committee 

November-December Data collection at six RESAs using focus group methodology 


2016 Activities 


January 

Data collection at the remaining two RESAs using focus group meth- 
odology 

February-June 

Ongoing data collection from catalyst school principals, including PLC 
schedules, PLC agendas, and SAI2 post-test 
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Appendix B. Interview Questions: Funding for Professional Learning 

April 15, 2016, Office of Federal Programs 


1 . How much of a district’s budget for professional learning is typically made up of Title 
II funds? 

2. What other funds are available for the support of professional learning? 

3. The High Quality Educator Committee initiated the Transforming Professional 
Learning project, one purpose of which was to decentralize responsibility and 
authority for professional learning and push both back to local decision-making — 
especially schools. 

4. Can you think of a pathway that would allow funds to be assigned directly to a 
school, or must they be requested and approved by local districts? 
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Appendix C. Interview Protocol and Questions for Focus Groups with 
Principals, November 2015- January 2016 

Informed Consent for Participation in a Focus Group as Part of the WVDE 
Evaluation Study of the Catalyst School Program 

By participating in this focus group, you are agreeing to take part in a research study. The 
purpose of the study is to evaluate the effectiveness of the catalyst school program. What 
we learn from this study may help improve this program and help guide planners in scaling 
up the program in the future. Your current participation in the study is limited to completing a 
survey (conducted separately) and participating in this focus group, which should not take 
more than 45 minutes of your time. You will be presented with a series of questions and 
asked for your views in response. 

Taking part in this study will put you at no more risk than you would experience during any 
normal day. Although you may not benefit directly by taking part in the study, it is possible 
that because of what we learn, the program may improve to better meet your needs or the 
needs of students. Your responses to this survey will be protected and will never be re- 
vealed as coming from you. All responses will be combined and reported as a group. 

You will receive no monetary or other reward for taking part in this research study. Partici- 
pating in the focus group is completely voluntary. If you decide not to take part or to stop at 
any time, there will be no penalties or loss of benefits to you. For more information about the 
catalyst school program, you may contact Teresa Hammond at the West Virginia Depart- 
ment of Education (phone 304-558-9994; email thammond@k1 2.wv.us . Additionally, if you 
have questions about this research study or the survey, you may contact Patricia Hammer 
(phone 304-558-7881 ; email )hammer@k1 2.wv.us) . 

This research study has been reviewed and approved by the West Virginia Department of 
Education (WVDE) Institutional Review Board (IRB). If you want to know more about the re- 
view of this study, you may contact the WVDE IRB co-chair, Andy Whisman (phone 304- 
558-7881 ; email swhisman@k1 2.wv.us) . 

Thank you for taking part in this important effort. 

Focus group interview questions (Principals) 

1 . What sorts of arrangements have you and your staff made for collaborative learning 
time? 

2. What other steps have you taken so far this school year toward becoming a Learning 
School? 

a. Using the SAI2 results 

b. Examining student data 

c. Identifying teacher learning needs 

3. How have you used, if at all, the SAI2 results in your planning? 

4. What sorts of supports, if any, has your school received from RESAs? 

5. What sorts of supports, if any, has your school received from your district in 
becoming a Learning School? 
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6. How well do you think the district central office staff understands the Learning School 
approach? 

7. How supportive is the district in having schools define their own learning needs? 

8. Have you had a chance to request specific training or technical assistance based on 
your teachers’ learning needs as they defined them? If so, who did you ask and how 
did that go? 

9. If you have been called upon by the district to implement a new program, were you 
able to engage in the professional learning it entailed within your PLCs to any 
extent? How did that work? 

10. What type of additional support do you need to improve the implementation of the 
Learning School approach? 

1 1 . What role do you see for your school in helping to implement the Learning School 
approach at other schools in your district or region? 

12. What would you change, if anything, about the catalyst school program? 

13. Is there anything else you would like to add? 
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Appendix D. Questions for Focus Groups With Cohort 1 Teachers, March - April 
2015 

Following informed consent (see Appendix C), the following questions were asked: 

1 . What does the Catalyst School program in your school involve? 

2. How has your role changed as a result of the implementation of the pilot? 

a. Have you taken on new responsibilities? 

b. How has the pilot affected other aspects of your work? 

3. What benefits are you seeing as a result of the Catalyst School program in your 
school? 

a. On students? 

b. Other school staff? 

c. Overall school climate? 

4. What part of the Catalyst School program is having the most positive effect so far? 

5. What challenges are you facing by participating in the Catalyst School program? 
What are your concerns? 

a. Is it specific to a particular strategy or general? 

6. Would you like to continue with the catalyst school program? 

7. What type of additional support do you need to improve the implementation of the 
program? 

8. What would you change about the Catalyst School program? 

9. Is there anything else you would like to add? 
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Appendix E. Implementation of the Transforming Professional Learning Initiative 


Fixsen and colleagues (2005) describe in their research synthesis of the literature on imple- 
mentation science, six stages of the implementation process: (a) exploration and adoption, 
(b) program installation, (c) initial implementation, (d) full operation, (e) innovation, and (f) 
sustainability. 

This account described the stages of implementation at the state level. 

The High Quality Educator Committee (HQE) of the West Virginia Board of Education 
(WVBE) formed the Transforming Professional Learning initiative in fall 2013, in response to 
calls for more effective use of the large sums of state and federal dollars being spent each 
year on professional development of teachers, seemingly with little to show for it. The initia- 
tive, funded by a grant from the Claude Worthington Benedum Foundation, was initially sup- 
ported by the National Commission on Teaching and America’s Future (NCTAF), which was 
charged with developing an action plan to redesign how professional learning was planned 
and implemented across the state. Since that time, the WVBE, West Virginia Department of 
Education (WVDE) and regional educational service agencies (RESAs) with additional sup- 
port from the professional learning association, Learning Forward, have gone through sev- 
eral stages of implementation, as described in the timeline below: 

Exploration and adoption 

September-December 2013 

• Procured planning grant from Claude Worthington Benedum Foundation 

• Contracted with NCTAF to develop action plan 

• Formed Transforming Professional Learning Advisory Committee, which 
included representatives of RESAs, Center for Professional Development, 
WVDE, WVBE, higher education, National Governors Association, Learning 
Forward, NCTAF, West Virginia Teacher of the Year, teachers’ unions, 
principals, district superintendents, and West Virginia Parent-Teacher 
Association 

• Planned two meetings of advisory committee for February and March of 2014 
January-March 2014 

• Convened two meetings of advisory group 

• Conducted statewide survey of teachers and principals to collect data about 
current views of professional learning experiences and professional learning 
needs 

• Developed white paper outlining four outcomes to be achieved and work 
groups to achieve outcomes: 

o Coherent and transparent (WVBE) 
o Goal-directed and improvement-oriented (Standards and 
Improvement Work Group) 
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o Quality, efficient, and balanced (Professional Development Work 
Group) 

o Needs-based, responsive, flexible, and inclusive (Stakeholder 
Engagement Work Group) 

April-May 2014 

• WVDE adopts Learning School approach as implementation model for 
Transforming Professional Learning initiative at the school level 

Program installation 

June-July 2014 

• Obtained approval for a coordinator position to oversee project at WVDE 

• Identified five schools to serve as catalyst schools, located in two RESAs, to 
pilot a new way for RESAs and schools to work together, using the Learning 
School approach developed by Learning Forward, based on the Learning 
Forward Standards for Professional Learning. 

• Contracted with Joellen Killion of Learning Forward to lead orientation 
sessions for catalyst schools. 

Initial implementation 

July-November 2014 

• Field two-day orientation with principals and leadership teams from five 
catalyst schools (three elementary, one middle school, and one high school) 

• RESA staff began ongoing work with catalyst schools to implement Learning 
School approach 

December 2014 

• Hired professional learning coordinator to oversee project in December 2014 

• Tasked the WVDE Office of Research, Accountability, and Data Governance 
to conduct an evaluation of the catalyst school project 

January-March 2015 

• Developed and obtained approval for evaluation plan, including logic model 
(OR) 

• Conducted data collection activities, including teacher and principal 
interviews, and Learning Forward’s Standards Assessment Inventory 2 
(SAI2) at all schools (OR) 

April-August 2015 

• Identified and invited a second cohort of 25 catalyst schools located in all 
eight RESAs 

• Administered SAI2 to new cohort as pretest and data source for them to use 
(short timeline, only partial participation) 
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• Provided critical feedback to Joellen Killion (Learning Forward) and 
professional development coordinator about first cohort orientation training, 
based on teacher and principal interviews 

• Based on feedback, held a separate, daylong orientation for RESA and 
district staff (from districts that had at least one catalyst school) and made 
other adjustments to training 

• Conducted 2-day orientation for Cohort 2 principals and teacher leaders in 
two sessions (northern and southern counties) 

• Professional learning coordinator transfers to a new position 

• Drafted and obtained approval for WVBE 201 5-201 6 Master Plan for 
Statewide Professional Development (PD Master Plan) to reflect statewide 
implementation of Transforming Professional Learning initiative, including 
developing structures and policies to support the Learning School approach 

• Selected members and convened first meeting of a new Professional 
Learning Advisory Council 

September-December 2015 

• Professional Learning Advisory Council drafts new professional learning 
policy (Policy 5500), which is based on supporting the PD Master Plan and 
objectives of the Transforming Professional Learning initiative 

• Data collection begins for implementation of the new PD Master Plan and 
Cohort 2 catalyst schools 
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Appendix F. Interview Questions for RESA Directors, Spring 2015 

1 . What are your general thoughts about the catalyst school program? 

a. Intent — teachers taking the lead in planning their own professional development 

b. Actual implementation 

2. What would you change about the catalyst school program next year? 

3. What sorts of assistance were you able to provide to the catalyst schools in your 
region? 

a. Training 

b. Technical assistance 

c. Resource procurement 

d. Other 

4. How did you work things out with the districts to coordinate your efforts? 

5. How has your role changed as a result of the implementation of the program? 

6. What part of the catalyst school program is having the most positive effect so far? 

7. How have you integrated or leveraged other related programs or initiatives? 

a. LDC or MDC 

b. School climate 

c. SPL/SPI approaches 

d. Others 

8. What challenges are you facing by participating in the program? What are your 
concerns? 

a. Cost 

b. Adequate number of staff 

c. Other 

9. What type of additional support do you think RESAs need to improve the 
implementation of the program? 

10. Is there anything else you would like to add? 
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Appendix G. Survey of Catalyst School Principals About PLC Scheduling 

Message to Catalyst School Principals 
Dear Catalyst School Leaders, 

It was a pleasure to meet with catalyst school principals and teacher leaders in the past few 
months. Thanks so much for taking the time to meet with me at your RESA and share your 
experiences. I have nearly finished going through the transcripts to discover themes about 
what sorts of supports you have received from RESA staff and others, as well as your rec- 
ommendations for improving the implementation experience for schools newly entering into 
the Learning School approach. I think we are getting a clearer picture about how to move 
forward with statewide implementation beginning next year. 

I have two requests: 

1 . Please let me know if and when you would like to have your teachers take the 
Standards Assessment Inventory 2 (SAI2) survey later this year. Taking the 
survey again will give you an indication of gains you have made in aligning 
professional learning at your school with the Learning School approach and the 
Standards for Professional Learning. This survey will be available next year, too, 
so even if you think you have not made as much progress as you would like, it 
may be useful to take the survey again this spring to set a baseline for next year. 
Survey results are strictly confidential. 

2. To give me more specific information about the array of learning communities in 
your school and how you carved out time for them, I have created a quick survey 
asking you to list PLCs in your building, and give me a few other quick pieces of 
information about each. The survey is the Word document attached. To fill it out, 
make sure you are in editing mode (not viewing), and where there are drop-down 
menus, just click on “Choose an item” and an arrow will appear to the right, which 
you can click on to select your response. I’m told this takes about 5-10 minutes to 
fill out. 

Thanks so much for taking the time! 

Best regards, 

Pat Hammer 

Patricia Cahape Hammer 

Research and Evaluation Specialist, IRB Chair 

WVDE Office of Research, Accountability, and Data Governance 

1900 Kanawha Boulevard, East 

Charleston, WV 25305-0330 

304.558.2546 P 

304.558.1613 F 

wvde. state. wv.us 
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School 

County 

Professional 
learning commu- 
nity or collabora- 
tive learning 
team name 

Number 
of mem- 
bers 

Usual frequency 
of meetings 
(select one) 

Usual meet- 
ing duration 
(minutes) 

When are meetings 
held? 

(select one) 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 



Choose an item. 


Choose an item. 

How did you free up time for meetings held during the regular wor 
an 8-hour day, student late arrival, etc.) 

< day? (e.g., adopting 

Is there anything else you would like to share about scheduling time for professional 
learning? 
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Appendix H. RESA 2 and RESA 6 Staff Questionnaire, Spring 2016 

1 . Approximately how many FTEs do you estimate you will have focused on supporting 
catalyst schools in your region between July 1 of 2015 and June 30 of 2016? 

2. Since the orientation last summer, how often have you met with your catalyst schools 
as a group? 

3. Since the orientation last summer, about how times and for what purpose have you 
visited each of the catalyst schools in your region? 


Name of school 

Approxi- 
mate num- 
ber of 

times 

Usual purpose(s) 























4. Since orientation last summer, please briefly describe any regular communications 
you may have had with district central office staff about the Learning School 
approach or the Standards for Professional Learning in the following settings: 

a. Regional meetings 

b. Individual districts 

c. Other 

5. Based on your experience with catalyst schools, 

a. Have you been able to integrate your work with catalyst schools with other 
programs — for example, focus schools? If so, please describe. 

b. Which aspects of becoming a Learning School have come most easily to the 
schools you have worked with? 

c. Which aspects have been most difficult for principals and teachers to commit 
to and incorporate into the way they approach professional learning? 

d. What is the range in the level of effort required to support implementation of 
the Learning School approach? 

o Time commitment for most responsive, adaptive schools? (frequency of 
visits, monthly time commitment) 

o Time commitment for least responsive, adaptive schools? (frequency of 
visits, monthly time commitment) 
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o Time commitment for more typical schools (frequency of visits, monthly 
time commitment) 

e. What do you project as the duration (number of years) of the intervention 
effort — either by RESA or trained district central office staff — that will be 
required for most schools? 

f. What are effective ways your RESA has developed for supporting districts’ 
work with their own catalyst schools? 

g. How do you see the role of RESAs in supporting implementation of the 
Learning School approach shifting during the next three years, if at all? 

h. Have you found ways to integrate the Standards for Professional Learning 
with the Standards for High Quality Schools, Teaching Standards, or principal 
standards? If so, please describe. 

6. What progress would you say you have made in helping districts and schools take 
advantage of the flexibility allowing in reimagined time? Have any of your counties 
applied for waivers? 

7. One of the goals of the Transforming Professional Learning initiative, was to allow 
the flow of funding for professional learning directly to schools so they can decide 
how to use it. 

a. What is your experience in entering into that conversation with district staff? 

b. Please describe any changes you may have seen in how the districts you 
have worked with have treated professional learning funding decisions? 

8. Do you have suggestions for accountability mechanisms to support implementation 
of the Standards for Professional Learning — for example in OEPA monitoring visits, 
principal or teacher evaluations? 

9. Do you have suggestions for some kind of statewide communications and 
recognition program? 

10. Do you have suggestions for how the WVDE might support the statewide 
implementation effort? 

a. Resources? (e.g., grants to support school participation in SAI2) 

b. Technical assistance or professional development 
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Appendix I. Category Codes for Activities Included in Catalyst School PLC 
Agendas 

Administrative information sharing 

• Logistics/schedules — Field trips, school events, bus schedules, etc. 

• Procedures — Management, GSA testing, WVEIS reporting, fire drills, and other 
related topics 

• News/info — Change from NCLB to ESSA, school board decisions, etc. 

• Other — Additional administrative topics 

Planning for learning 

• Student learning — Reviewing individual and group-level student academic and early 
warning data, making adjustments, monitoring progress, planning PBL projects, and 
other related topics 

• Professional learning — Planning for teacher learning needs based on student 
needs and based on educator evaluation, IPI, SAI2, OEPA data and other data 

• Other planning for learning — Additional planning topics 

Engaging in professional learning about . . . 

• Student behavior/school climate/parent involvement — PBIS, school climate and 
culture, parent involvement, attendance matters, using the early warning data, and 
other related topics 

• Content/pedagogy — Content area, instruction, formative assessment, how to use 
student data, using new curriculum materials, etc. 

• Professional learning systems — Becoming a Learning School, effective PLCs, IPI 
training, etc. 

• Other — Professional development on other topics 
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Appendix K. Rapid Identification of Themes From Audio Recordings (RITA) Coding Sheet for Principal Focus Group 
Interviews 



Coding form for principal interviews - Excel 

FORMULAS DATA REVIEW VIEW DEVELOPER ADD-INS Acrobat 

i E = = o I- 0 *i a, i- I ri i EB * 
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1 

1 






















LEARNING COMMUNITIES-committed to 

11 

26.8 






















Continuous improvement 







1 




1 










1 




Collective responsibility 






1 

1 




1 








1 

1 





Goal alignment 




1 

1 















1 




LEARNING DESIGNS 

3 

7.3 






















Aligns with teacher experience/needs 







1 


















0 

Uses variety of forms 




1 





















1 

Uses peer observation 



1 






















2 

OUTCOMES 

5 

12.2 



















3 

Explicitly links educator and student learning 















1 


1 





1 




4 

Focuses on curriculum and how students learn 




1 


















1 




5 

Builds on previous teacher learning 


























6 

RESOURCES 

5 

12.2 


7 

Allocates time for PL during school day 



1 

1 

















1 




3 

Expenditures for PL openly discussed 

























9 

Provides technology and other PL resources 
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Appendix L. Message to RESA Directors Request Stage-of-lmplementation for 
Catalyst Schools 

Friday, June 3, 2016 

Subject: Quick assessment of the catalyst schools in your region 
Dear RESA Director, 

Which of the following necessary stages do you think each of your catalyst schools have 
reached in implementing the Learning School approach? NOTE: Schools, depending on 
their prior experiences and culture, will move at their own pace, so there could be a wide 
range in your region. I will use your ratings in combination with several other data points I 
have collected for each school. As always your responses will be kept confidential. 

Definition of stages (adapted from Fixsen et al., 2005) 

1 . Exploration — identifying the need, acquiring information, assessing the fit 
between the need and intervention program, and preparing the organization by 
mobilizing information and support 

2. Installation — making preparations, such as human resource strategies, policy 
development, reporting frameworks, outcome expectations, staff training, and 
needed technology/resources 

3. Initial implementation — beginning to use and monitor new practices, 
developing and applying new skills, creating a supportive organizational culture, 
overcoming inertia and resistance 

4. Full implementation — proceeding with innovation as accepted practice and 
routine, and seeing evidence of expected impacts/improvements 

5. Sustainability — maintaining the effectiveness of the innovation as staff come 
and go, and other changes that take place in the organization and its 
environment 


Please place an “X” in the appropriate column for each school. 


Name of school 

Exploration 

1 

Installation 

2 

Initial imple- 
mentation 

3 

Full imple- 
mentation 

4 

Sustainability 

5 


















































Thanks for this quick assessment! 
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